Subscribe to release notesStatus pageCommunity
Developer guides

Import prompt and response annotations

Developer guide for importing annotations on prompt and response projects.

Overview

To import annotations in Labelbox, you need to create an annotations payload. In this section, we provide this payload for every supported annotation type.

Annotation payload types

Labelbox supports two formats for the annotations payload:

  • Python annotation types (recommended)
    • Provides a seamless transition between third-party platforms, machine learning pipelines, and Labelbox.
    • Allows you to build annotations locally with local file paths, numpy arrays, or URLs.
    • Supports easy conversion to NDJSON format to quickly import annotations to Labelbox.
    • Supports one-level nested classification (radio, checklist, or free-form text) under a tool or classification annotation.
  • JSON
    • Skips formatting annotation payload in the Labelbox Python annotation type.
    • Supports any levels of nested classification (radio, checklist, or free-form text) under a tool or classification annotation.

Label Import Types

Labelbox supports two types of label imports:

  • Model-assisted labeling (MAL) allows you to import computer-generated predictions and simple annotations created outside of Labelbox as pre-labels on an asset.
  • Ground truth allows you to bulk import ground truth annotations from an external or third-party labeling system into Labelbox Annotate. Using the label import API to import external data can consolidate and migrate all annotations into Labelbox as a single source of truth.

Supported Annotations

Prompt and response generated projects support the following annotations data row:

  • Prompt and response creation projects

    • Prompt text
    • Radio
    • Checklist
    • Response text

  • Prompt creation projects

    • Prompt text

  • Response creation projects

    • Radio
    • Checklist
    • Response text

Prompt

Free-form text

📘

Information

Only one prompt annotation is allowed per label.

prompt_annotation = lb_types.PromptClassificationAnnotation(
    name = "Follow the prompt and select answers",
    value = lb_types.PromptText(answer = "This is an example of a prompt")
)

prompt_annotation_ndjson = {
  "name": "Follow the prompt and select answers",
  "answer": "This is an example of a prompt"
}

Responses

Radio (single choice)

response_radio_annotation = lb_types.ClassificationAnnotation(
    name="response radio feature",
    value=lb_types.Radio(answer = 
        lb_types.ClassificationAnswer(name = "first_radio_answer")
    )
)

response_radio_annotation_ndjson = {
  "name": "response radio feature",
  "answer": {
      "name": "first_radio_answer"
    }
}

Checklist (multiple choice)

response_checklist_annotation = lb_types.ClassificationAnnotation(
    name="response checklist feature",
    value=lb_types.Checklist(answer = [
        lb_types.ClassificationAnswer(name = "option_1"),
        lb_types.ClassificationAnswer(name = "option_2"),
    ])
  )

response_checklist_annotation_ndjson = {
  "name": "response checklist feature",
  "answer": [
    {
      "name": "option_1"
    },
    {
      "name": "option_2"
    }
  ]
}

Free-form text

response_text_annotation = lb_types.ClassificationAnnotation(
    name = "Provide a reason for your choice", 
    value = lb_types.Text(answer = "This is an example of a response text")
)

response_text_annotation_ndjson = {
  "name": "Provide a reason for your choice",
  "answer": "This is an example of a response text"
}

Nested classification

nested_response_radio_annotation = lb_types.ClassificationAnnotation(
  name="nested_response_radio_question",
  value=lb_types.Radio(
    answer=lb_types.ClassificationAnswer(
      name="first_radio_answer",
      classifications=[
        lb_types.ClassificationAnnotation(
          name="sub_radio_question",
          value=lb_types.Radio(
            answer=lb_types.ClassificationAnswer(
              name="first_sub_radio_answer"
            )
          )
        )
      ]
    )
  )
)

nested_response_checklist_annotation = lb_types.ClassificationAnnotation(
  name="nested_response_checklist_question",
  value=lb_types.Checklist(
    answer=[lb_types.ClassificationAnswer(
      name="first_checklist_answer",
      classifications=[
        lb_types.ClassificationAnnotation(
          name="sub_checklist_question",
          value=lb_types.Checklist(
            answer=[lb_types.ClassificationAnswer(
            name="first_sub_checklist_answer"
          )]
        ))
      ]
    )]
  )
)

nested_response_radio_annotation_ndjson = {
  "name": "nested_radio_question",
  "answer": [{
      "name": "first_radio_answer", 
      "classifications" : [
        {
          "name": "sub_radio_question", 
          "answer": {"name": "first_sub_radio_answer"}
        }          
      ]         
  }]
}

nested_response_checklist_annotation_ndjson = {
  "name": "nested_checklist_question",
  "answer": [{
      "name": "first_checklist_answer", 
      "classifications" : [
        {
          "name": "sub_checklist_question", 
          "answer": {"name": "first_sub_checklist_answer"}
        }          
      ]         
  }]
}

Example: Import pre-labels or ground truths

The steps to import annotations as pre-labels (machine-assisted learning) are similar to the steps to import annotations as ground truth labels, and we will describe the slight differences for each scenario.

Before you start

The below imports are needed to use the code examples in this section.

import labelbox as lb
import labelbox.types as lb_types
import uuid
import time

Replace the value of API_KEY with a valid API key to connect to the Labelbox client.

API_KEY = None
client = lb.Client(API_KEY)

Step 1: Import or generate data rows and create projects

Each type of the prompt and response generation project requires different setup. See prompt and response project for more details on the differences.

Prompt response and prompt creation

A prompts and responses creation project automatically generates empty data rows upon creation. You will then need to obtain either the global_keys or data_row_ids attached to the generated data rows by exporting them from the created project or obtaining them directly on the data row tab using the UI.

prompt_response_project = client.create_model_evaluation_project(
    name="Demo prompt response project",
    media_type=lb.MediaType.LLMPromptResponseCreation,
    dataset_name="Demo prompt response dataset",
    data_row_count=1,
)

prompt_project = client.create_model_evaluation_project(
    name="Demo prompt project",
    media_type=lb.MediaType.LLMPromptCreation,
    dataset_name="Demo prompt dataset",
    data_row_count=1,
)

Response creation

For response creation projects, text data rows are used and are not generated upon project creation. The following steps create a dataset with a text data row attached, create a response creation project, and batch the created data row towards the project.

# Create dataset with text data row
global_key = "lorem-ipsum.txt"
text_asset = {
    "row_data": "https://storage.googleapis.com/labelbox-sample-datasets/nlp/lorem-ipsum.txt",
    "global_key": global_key,
    "media_type": "TEXT",
    }

dataset = client.create_dataset(name="text_annotation_import_demo_dataset")
task = dataset.create_data_rows([text_asset])
task.wait_till_done()
print("Errors:",task.errors)
print("Failed data rows:", task.failed_data_rows)

# Create response creation project
project = client.create_response_creation_project(
    name="Demo response project",
)

# Create a batch of data rows for newly created project
batch = project.create_batch(
    "Demo response batch",  # each batch in a project must have a unique name
    global_keys=[global_key], # paginated collection of data row objects, list of data row ids or global keys
    priority=1  # priority between 1(highest) - 5(lowest)
)

Step 2: Set up ontology

Your project ontology needs to support the classifications required by your annotations. To ensure accurate schema feature mapping, the value used as the name parameter needs to match the value of the name field in your annotation.

For example, if you provide a name annotation_name for your created annotation, you need to name the bounding box tool as anotations_name when setting up your ontology. The same alignment must hold true for the other tools and classifications that you create in the ontology.

This example shows how to create an ontology containing all supported by prompt and response project types annotation types.

ontology_builder = lb.OntologyBuilder(
    tools=[],
    classifications=[
        lb.PromptResponseClassification(
            class_type=lb.PromptResponseClassification.Type.PROMPT,
            name="prompt text",
            character_min = 1, # Minimum character count of prompt field (optional)
            character_max = 20, # Maximum character count of prompt field (optional)
        ),
        lb.PromptResponseClassification(
            class_type=lb.PromptResponseClassification.Type.RESPONSE_CHECKLIST,
            name="response checklist feature",
            options=[
                lb.ResponseOption(value="option_1", label="option_1"),
                lb.ResponseOption(value="option_2", label="option_2"),
            ],
        ),
        lb.PromptResponseClassification(
            class_type=lb.PromptResponseClassification.Type.RESPONSE_RADIO,
            name="response radio feature",
            options=[
                lb.ResponseOption(value="first_radio_answer"),
                lb.ResponseOption(value="second_radio_answer"),
            ],
        ),
        lb.PromptResponseClassification(
            class_type=lb.PromptResponseClassification.Type.RESPONSE_TEXT,
            name="response text",
            character_min = 1, # Minimum character count of response text field (optional)
            character_max = 20, # Maximum character count of response text field (optional)
        ),
        lb.PromptResponseClassification(
            class_type=lb.PromptResponseClassification.Type.RESPONSE_RADIO,
            name="nested_response_radio_question",
            options=[
                lb.ResponseOption("first_radio_answer",
                                  options=[
                                    lb.PromptResponseClassification(
                                      class_type=lb.PromptResponseClassification.RESPONSE_RADIO,
                                      name="sub_radio_question",
                                      options=[lb.ResponseOption("first_sub_radio_answer")]
                                    )
                                  ])
            ],
        ),
        lb.PromptResponseClassification(
            class_type=lb.PromptResponseClassification.Type.RESPONSE_CHECKLIST,
            name="nested_response_checklist_question",
            options=[
                lb.ResponseOption("first_checklist_answer",
                                  options=[
                                    lb.PromptResponseClassification(
                                      class_type=lb.PromptResponseClassification.RESPONSE_CHECKLIST,
                                      name="sub_checklist_question",
                                      options=[lb.ResponseOption("first_sub_checklist_answer")]
                                    )
                                  ])
            ],
        ),
    ],
)

# Create ontology
ontology = client.create_ontology(
    "Prompt and response ontology",
    ontology_builder.asdict(),
    media_type=lb.MediaType.LLMPromptResponseCreation,
)

# Connect ontology 
prompt_response_project.connect_ontology(ontology)

ontology_builder = lb.OntologyBuilder(
    tools=[],
    classifications=[
        lb.PromptResponseClassification(
            class_type=lb.PromptResponseClassification.Type.PROMPT,
            name=f"prompt text",
            character_min = 1, # Minimum character count of prompt field (optional)
            character_max = 20, # Maximum character count of prompt field (optional)
        )
    ],
)

# Create ontology
ontology = client.create_ontology(
    "Prompt ontology",
    ontology_builder.asdict(),
    media_type=lb.MediaType.LLMPromptCreation,
)

# Connect ontology 
prompt_project.connect_ontology(ontology)

ontology_builder = lb.OntologyBuilder(
    tools=[],
    classifications=[
        lb.PromptResponseClassification(
            class_type=lb.PromptResponseClassification.Type.RESPONSE_CHECKLIST,
            name="response checklist feature",
            options=[
                lb.ResponseOption(value="option 1", label="option 1"),
                lb.ResponseOption(value="option 2", label="option 2"),
            ],
        ),
        lb.PromptResponseClassification(
            class_type=lb.PromptResponseClassification.Type.RESPONSE_RADIO,
            name="response radio feature",
            options=[
                lb.ResponseOption(value="first_radio_answer"),
                lb.ResponseOption(value="second_radio_answer"),
            ],
        ),
        lb.PromptResponseClassification(
            class_type=lb.PromptResponseClassification.Type.RESPONSE_TEXT,
            name="response text",
            character_min = 1, # Minimum character count of response text field (optional)
            character_max = 20, # Maximum character count of response text field (optional)
        ),
        lb.PromptResponseClassification(
            class_type=lb.PromptResponseClassification.Type.RESPONSE_RADIO,
            name="nested_response_radio_question",
            options=[
                lb.ResponseOption("first_radio_answer",
                                  options=[
                                    lb.PromptResponseClassification(
                                      class_type=lb.PromptResponseClassification.RESPONSE_RADIO
                                      name="sub_radio_question",
                                      options=[lb.ResponseOption("first_sub_radio_answer")]
                                    )
                                  ])
            ],
        ),
        lb.PromptResponseClassification(
            class_type=lb.PromptResponseClassification.Type.RESPONSE_CHECKLIST,
            name="nested_response_checklist_question",
            options=[
                lb.ResponseOption("first_checklist_answer",
                                  options=[
                                    lb.PromptResponseClassification(
                                      class_type=lb.PromptResponseClassification.RESPONSE_CHECKLIST
                                      name="sub_checklist_question",
                                      options=[lb.ResponseOption("first_sub_checklist_answer")]
                                    )
                                  ])
            ],
        ),
    ],
)

# Create ontology
ontology = client.create_ontology(
    "Response ontology",
    ontology_builder.asdict(),
    media_type=lb.MediaType.Text,
    ontology_kind=lb.OntologyKind.ResponseCreation
)

# Connect ontology 
response_project.connect_ontology(ontology)

Step 3: Export for global_keys

For prompt response creation and prompt creation projects you will need to obtain either the global_keys or data_row_ids attached to the generated data rows by exporting them from the created project. Since the generation of data rows is an async process you will need to wait for the project data rows to be completed before exporting.

time.sleep(20)

export_task = prompt_response_project.export()
export_task.wait_till_done()


# Check export for any errors
if export_task.has_errors():
  export_task.get_buffered_stream(
  stream_type=lb.StreamType.ERRORS
  ).start(stream_handler=lambda error: print(error))

stream = export_task.get_buffered_stream()

# Obtain global keys to be used later on
global_keys = [dr.json["data_row"]["global_key"] for dr in stream]

time.sleep(20)

export_task = prompt_project.export()
export_task.wait_till_done()


# Check export for any errors
if export_task.has_errors():
  export_task.get_buffered_stream(
  stream_type=lb.StreamType.ERRORS
  ).start(stream_handler=lambda error: print(error))

stream = export_task.get_buffered_stream()

# Obtain global keys to be used later on
global_keys = [dr.json["data_row"]["global_key"] for dr in stream]

Step 4: Import Annotations

For prelabeled (model-assisted labeling) scenarios, pass your payload as the value of the predictions parameter. For ground truths, pass the payload to the labels parameter.

Payload

Depending on the type of prompt and response project you are using your payload might look different. For the response payload, you can also use is_benchmark_reference to specify benchmarks.

# Python annotation objects
label = []
annotations = [
  prompt_annotation,
  response_radio_annotation,
  response_checklist_annotation,
  response_text_annotation,
  nested_response_radio_annotation,
  nested_response_checklist_annotation
]
label.append(
    lb_types.Label(data={"global_key" : global_keys[0] },
                   annotations=annotations)
            )

# NDJSON
label_ndjson = []
annotations = [
  prompt_annotation_ndjson,
  response_radio_annotation_ndjson,
  response_checklist_annotation_ndjson,
  response_text_annotation_ndjson,
  nested_response_radio_annotation_ndjson,
  nested_response_checklist_annotation_ndjson
]
for annotation in annotations:
    annotation.update({
        "dataRow": {
            "globalKey": global_keys[0]
        },
    })
    label_ndjson.append(annotation)

# Python annotation objects
label = []
annotations = [
  prompt_annotation
]
label.append(
    lb_types.Label(data={"global_key" : global_keys[0] },
                   annotations=annotations)
            )

# NDJSON
label_ndjson = []
annotations = [
  prompt_annotation_ndjson
]
for annotation in annotations:
    annotation.update({
        "dataRow": {
            "globalKey": global_keys[0]
        },
    })
    label_ndjson.append(annotation)

# Python annotation objects
label = []
annotations = [
  response_radio_annotation,
  response_checklist_annotation,
  response_text_annotation,
  nested_response_radio_annotation,
  nested_response_checklist_annotation
]
label.append(
    lb_types.Label(data={"global_key" : global_key },
                   annotations=annotations
                   # Optional: set the label as a benchmark
                   # Only supported for groud truth imports
                   is_benchmark_reference = True)
            )

# NDJSON
label_ndjson = []
annotations = [
  response_radio_annotation_ndjson,
  response_checklist_annotation_ndjson,
  response_text_annotation_ndjson,
  nested_response_radio_annotation_ndjson,
  nested_response_checklist_annotation_ndjson
]
for annotation in annotations:
    annotation.update({
        "dataRow": {
            "globalKey": global_key
        },
    })
    label_ndjson.append(annotation)

Option A: Upload as prelabels (model assisted labeling)

This option is helpful for speeding up the initial labeling process and reducing the manual labeling workload for high-volume datasets.

# Upload MAL label for this data row in project
upload_job = lb.MALPredictionImport.create_from_objects(
    client = client, 
    project_id = prompt_response_project.uid, # Replace with project of different prompt and response project types
    name = "mal_job"+str(uuid.uuid4()), 
    predictions = label
)

print(f"Errors: {upload_job.errors}", )
print(f"Status of uploads: {upload_job.statuses}")

Option B: Upload to a labeling project as ground truth

This option is helpful for loading high-confidence labels from another platform or previous projects that just need review rather than manual labeling effort.

# Upload label for this data row in project
upload_job = lb.LabelImport.create_from_objects(
    client = client, 
    project_id = prompt_response_project.uid, # Replace with project of different prompt and response project types
    name = "label_import_job" + str(uuid.uuid4()),  
    labels = label
)

print(f"Errors: {upload_job.errors}", )
print(f"Status of uploads: {upload_job.statuses}")